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With increased accountability attached to students' results on national testing in 
Australia, teachers feel under pressure to prepare students for the tests. However, 
this can lead to shallow teaching of a narrowed curriculum. An alternative approach 
involves using data to identify common errors and misconceptions, discussing 
strategies aimed at building understanding of important mathematical ideas as well 
as students' confidence in answering context-based mathematics questions. This 
study explored the use of a learning model based on professional conversations 
about national testing results as well as school-based assessment data with junior 
secondary mathematics teachers in one school. The teachers identified the learning 
needs of students and chose to implement mental computation and estimation 
approaches as well as a strategy to address the literacy demands of numeracy test 
items to support student learning before and after the NAPLAN test. An analysis of 
the professional learning model identified approaches to enhance both student 
learning and teaching practice. 


Background 


In Australia, the debate surrounding mathematics and numeracy achievement 
has been similar to that experienced elsewhere. There is a growing recognition of 
the need for greater numeracy proficiency and that early intervention provides 
the best chance of success for children at risk of failure. The concern about 
numeracy by Australian governments was first highlighted in the National 
Literacy and Numeracy Plan (DETYA, 2000), which provided a framework for 
improving the literacy and numeracy outcomes of all students. This plan 
embraced the development of the national benchmarks for students in Years 3, 5 
and 7, as well as the need for assessment and reporting against these 
benchmarks. Until recently, each state and territory in Australia collected student 
achievement data for the Federal Government. Concern about the proportion of 
students not meeting the minimum national benchmark standards (Curriculum 
Corporation, 2000) has continued with large investments by governments to 
address the needs of students at risk. 

To better standardise the monitoring of student achievement the National 
Assessment Program in Literacy and Numeracy (NAPLAN) was introduced in 
2008 (DETYA, 2000). The same tests in literacy and numeracy are now 
administered nationally to all students in Years 3, 5, 7 and 9. Testing early in the 
school year potentially provides diagnostic information to teachers about their 
students' performance in mathematics topics common to all states and territories 
(Curriculum Corporation, 2006). 

Whether we approve of a national testing regime or not, this level of 
accountability is in place for the foreseeable future with pressure on school 
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principals and teachers to improve results. While the information may be useful 
after the results are released, teachers of Years 3, 5, 7 and 9 are experiencing 
increased pressure early in the school year to prepare students for the test. 
Principals, school systems personnel and parents are scrutinising the results to 
determine whether schools and their teachers are 'measuring up'. Public 
comparisons between 'statistically similar' schools are now possible with the 
Federal Government sponsored My School website which presents statistical and 
contextual information about schools. 

The results from the NAPLAN assessments are reported in individual 
student reports to parents, as well as school and aggregate reports with 
substantial information including results for each item and for each student. The 
school reports enable teachers to analyse the results for each year group to deter¬ 
mine which items appear to be understood and which are problematic. In addition, 
school data can be compared to the Australian student data. The information is 
useful to address common errors and misconceptions as well as to aid planning 
and programming of future learning (Perso, 2009). Rather than abandon good 
pedagogical practices and have students individually practise test items, 
NAPLAN items can be used to address key issues in students' understanding 
and develop appropriate quality-teaching approaches (Anderson, 2009). 

The purpose of the project reported here was to engage teachers in 
professional learning conversations about using evidence from their own 
NAPLAN results to identify their students' needs and collaboratively develop 
pedagogical practices which research has shown to be beneficial in building 
understanding. In particular, this paper describes and analyses the outcomes of 
a professional learning program conducted in one school by addressing the 
following research questions. 

1. What strategies did teachers choose to use to support student 
preparation for NAPLAN and how was this different to previous 
practice? 

2. Did the professional learning support have an impact on student 
learning and on teaching practice (including attitudes)? 

Literature Review 


Teaching to the Test! 

High-stakes testing has been criticised for encouraging teachers to limit the 
curriculum to what is assessed (Abrams, Pedulla & Madaus, 2003) and resulting 
in the "corruption of indicators and educators" (Nichols & Berliner, 2005, p. 1). 
While the types of testing being conducted in some states in the United States of 
America in recent years could be considered higher stakes than the NAPLAN 
testing in Australia, systems, principals and teachers feel under pressure to 
prepare students for the tests and achieve good results, particularly given the 
publishing of data on the My School website. The pressure to raise scores has the 
potential to distort teaching and learning but there are ways teachers can support 
students' preparation for high-stakes tests without detracting from real learning 
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(Gulek, 2003). Miyasaka (2000) identified five types of test preparation practices 
that support student learning and improve achievement - teaching the 
mathematics content, using a variety of assessment approaches, teaching time 
management skills with practise in test-taking, reviewing and assessing content 
throughout the year, as well as fostering student motivation and reducing test 
anxiety. In addition, Marzano, Kendall and Gaddy (1999) found knowledge of 
test vocabulary and terminology improves student performance. 

Compulsory testing of students in Years 3, 5, 7 and 9 in Australia has the 
potential to focus teachers' efforts on preparing students for the test by using 
past papers for practise and limiting learning to technical support such as how 
to fill in answers (Nisbet, 2004). However, balancing this narrow approach is the 
potential benefit of identifying students' strengths and weaknesses with data 
informing planning and teaching. In a survey of 56 primary schools, Nisbet 
(2004) reported two thirds of the schools in his study used data to identify topics 
causing difficulties but only 40% of teachers used the results to identify 
individual students who were having difficulty. Further, only 22% used the 
results to plan their teaching. The low proportion of primary school teachers 
using the data to inform teaching and learning represents a missed opportunity 
and there is little evidence that secondary mathematics teachers are analysing 
NAPLAN data in meaningful ways. 

An Alternative Approach - Engaging Teachers in Professional 
Conversations about Data 

There is an alternative approach to 'teaching to the test' but the evidence above 
suggests teachers require support to analyse and interpret the data and consider 
alternative practices, to address common student misconceptions and difficulties 
(Anderson, 2009). Gulek (2003, p. 42) refers to the need for "school practitioners 
to become assessment literate in order to make the maximum use of test results" 
and Thomson and Buckley (2009) describe the potential of test item analysis to 
inform pedagogy. It should be noted the test preparation practices that we are 
advocating are aimed at improving students' knowledge, skills and 
understanding of numeracy and mathematics and not at artificially increasing 
students' test scores. Unlike Dimarco (2009) who criticises giving any attention to 
such tests, we believe teachers' professional standing does not need to be 
compromised by considering how NAPLAN items can be used to improve 
student learning. 

Planning professional learning opportunities for teachers in relation to new 
assessment regimes, or new approaches to teaching and learning, requires 
consideration of several factors which impact on teachers' practice in classrooms 
such as teachers' knowledge, beliefs and attitudes (Wilson & Cooney, 2002). 
Rather than change in beliefs and attitudes preceding change in practice, 
Guskey's (2002) model of teacher change proposes professional learning 
precedes the implementation of new ideas in classrooms, which when 
implemented could lead to a positive change in student learning outcomes, and 
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subsequently, a change in teachers' beliefs and attitudes. This model suggests 
that teachers need to try new ideas and witness positive student outcomes before 
they fully embrace such approaches. 

Following Earl and Timperley's (2009) research into the use of evidence to 
inform practice and building on Guskey's (2002) model of teacher change, the 
professional learning model developed for this project aimed to engage 
secondary mathematics teachers' in rich conversations about data including 
NAPLAN and whether NAPLAN items provide opportunities for learning and 
teaching. As noted by Earl and Timperley these conversations required more 
than just looking at their students' results. 

... conversations that are grounded in evidence and focused on learning from 
that evidence have considerable potential to influence what happens in schools 
and ultimately enhance the quality and the efficiency of student learning. We 
have also come to the conclusion that having conversations based on data in 
educational contexts is very hard to do. It is hard because productive use of 
evidence requires more than just adding data to the conversation; it involves a 
way of thinking and challenging ideas towards new knowledge, (p. 2) 

The research design was based on a model of "productive evidence-based 
conversations" (Earl & Timperley, 2009, p. 3), which has particular qualities (see 
Figure 1). The conversations involve having an "inquiry habit of mind", with 
discussions about a range of relevant evidence where relationships are respectful 
but allowing for challenge. The approach taken in this study involved a group of 
teachers from the same school discussing the evidence from the previous 
NAPLAN Numeracy test for their students, asking questions about the data 
informed by classroom-based knowledge of their students, identifying topic 
areas requiring further investigation, and developing strategies to address the 
particular learning needs of their students. 


Using 

Relevant 

Data 



Figure 1. Processes for evidence-informed conversations (Earl & Timperley, 

2009, p. 3) 











64 


Paul White & Judy Anderson 


Pedagogical Approaches to Improve Students' Engagement with 
Context-based Mathematics Questions 

National testing agendas can provide an opportunity if we use test items to assist 
students who have difficulty reading and interpreting mathematical text, to 
further develop students' thinking skills, and to analyse common errors and 
misconceptions, frequently presented as alternative solutions in multiple-choice 
items. One practical approach to 'teaching to the test' while maintaining sound 
pedagogical practices is to use NAPLAN items as discussion starters so that 
students develop number sense, adopt new problem-solving strategies, and 
build confidence and resilience. Hence, teachers' professionalism need not be 
compromised by national testing agendas as long as they adopt teaching 
strategies, which use the data in meaningful ways to inform their planning and 
teaching. 

Research has advocated several teaching practices that have the potential to 
target particular aspects of students' difficulties in mathematics and numeracy. 
While many strategies could be considered, in this project, the following 
strategies were chosen based on sources of students' errors; mental computation, 
estimation and number sense; and the literacy demands of context-based 
mathematics questions. 

Common student misconceptions have been identified as a major source of 
errors. For example, Ryan and Williams (2007, p. 23) use the term "intelligent 
overgeneralization" to refer to students' predisposition to create inappropriate 
rules based on experiences. Some common generalisations include: 
multiplication makes bigger; division makes smaller; division is necessarily of a 
bigger number by a smaller number; and longer numbers are always greater in 
value. Figure 2 presents a NAPLAN Numeracy item where this type of over¬ 
generalisation occurs with few students selecting the correct answer of 22. 


What is the answer to 6.6 + 0.3? 

A) 0.022 B) 0.22 C) 2.2D) 22 


Figure 2. An item from the 2008 Year 7 non-calculator allowed numeracy 

NAPLAN test 

A common fraction misconception occurs when area is not the feature students 
identify in regional models of fractions (Gould, Outhred & Mitchelmore, 2006). 
The "number of pieces" interpretation is a common response. This research 
explains the responses to the 2008 Year 7 NAPLAN item shown in Figure 3 where 
only 28% correctly selected the last option. The fact that three parts (though 
unequal) were shaded obviously prompted most students to see it representing 
three quarters. The most popular response was option c. 
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Figure 3. A fraction item from the 2008 Year 7 non-calculator numeracy 

NAPLAN test. 

1. Mental Computation, Estimation and Number Sense 

In dealing with misconceptions like these, Anderson (2009) points out that 
encouraging students to apply reasoning about numbers to evaluate answers can 
be a challenge. She argues that one way to support the development of students' 
thinking strategies is to use test items that focus on mental computation, 
estimation and number sense (McIntosh, Reys & Reys, 1997). While students are 
frequently reluctant to estimate, this is an important first step. Options in 
multiple-choice items may often be eliminated after considering whether the 
solutions are reasonable. Anderson proposes that after students have estimated 
the answer teachers can pose questions such as the following. 

1. What strategies could you use to check the solution? 

2. What would the question need to be to obtain each of the alternative 
answers? 

3. What happens when you multiply a whole number by a number less 
than one? 

An estimation focus allows test items to provide a source of meaningful 
mathematical discussion. 

2. Literacy Demands of Context-based Mathematics Questions 

The contextual nature of many NAPLAN items and the associated language 
implications often lead to claims that these tests are more comprehension than 
mathematics. However, interpreting mathematical situations in context is what 
numeracy is all about. Hence, we claim the contextual nature of the items is at 
the heart of numeracy and deserving of special attention. Further, it seems 
pointless to pursue repetitive symbolic manipulation exercises to address poor 
responses to contextual items. 

Newman (1983) developed an error analysis protocol to analyse student 
responses to contextual items. She identified five levels of difficulty (Table 1). 
Most errors occurred in the second and third levels of 'comprehending' and 
'transforming' the text into an appropriate mathematical strategy, not applying 
the symbolic procedure. By translating each of the levels from Table 1 into a 
question for students, teachers are able to determine their first level of difficulty 
(White, 2005). 
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Table 1 


Levels in Newman's Error Analysis 


Reading the question 
Comprehending what is read 

Transforming the words into an appropriate 
mathematical strategy 

Applying the mathematical process skills 
Encoding the answer into an acceptable form 


Reading 

Comprehending 

Transforming 


Processing 

Encoding 


Methodology 


One school with a high NESB enrolment and a history of NAPLAN results below 
state and national average volunteered to participate in the project. Ten teachers 
of Years 7 and 9 (12 classes in total) were involved. The Professional Learning 
Model had five stages. 

Stage 1 involved teachers collecting data about their own students' ability in 
NAPLAN style items. In May each year. Years 7 and 9 students complete two 32- 
item test papers for Numeracy, one with and one without the use of a calculator. 
With the teachers, the authors used the 2008 NAPLAN numeracy test results for 
the school to identify specific areas of the curriculum requiring review and 
consolidation. Items from the 2008 NAPLAN papers in these areas were used to 
compile a short pre-test for diagnostic purposes for each of Years 7 and 9 
consisting of 5 non-calculator and 5 calculator items. Though the results from 
2008 were those of the current Year 8 and 10, not the cohorts involved in the 
project, they were still considered reflective of teaching approaches in the school 
because the teachers were the same. Furthermore, the value of the selected items 
would be gauged by how the students responded to them. 

Teachers administered the tests in early March, slightly more than two 
months before the NAPLAN tests in May 2009. Each teacher corrected their class 
responses to reveal the number of students selecting each option in multiple- 
choice items or the common solutions to the tree-response items. In the six Year 
7 classes, only one class had more than 50% of total responses correct in the 
calculator and non-calculator pre-tests (same class). In the six Year 9 classes, two 
had more than 50% of total responses correct in the non-calculator pre-test and 
no class had more than 50% of total responses correct in the calculator pre-test. 
These data support the items chosen as being areas of difficulty for the students. 

Stage 2 involved a one-day meeting (two months before the NAPLAN tests) 
between the teachers and the authors. The day consisted of professional learning 
conversations to review the students' pre-test responses, consider the key 
mathematical ideas and misconceptions in the tasks, compare this to data 
collected using school-based assessment procedures, and explore a range of 
possible research-based teaching approaches identified by the authors. Teachers 
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were encouraged to pose questions about the data. They also contributed 
suggestions about the mathematical issues they saw as relevant and strategies 
they believed could be used to address the identified student difficulties. From 
these conversations, a list of possible strategies was jointly constructed. Each 
teacher then nominated one or more to implement in their general teaching as 
well as specifically with targeted NAPLAN items. 

Stage 3 where teachers implemented their chosen strategies occurred over 
the next two months prior to the NAPLAN tests and continued beyond the tests. 
In this stage, lesson observations by a trained research assistant who is a 
qualified mathematics teacher were conducted. 

Stages 4 and 5 involved further professional learning conversations about 
the effectiveness and learning from the project in October 2009 and September 
2010. Data collected in Stage 4 involved teacher questionnaires and interviews 
from the original 10 teachers and in Stage 5, eight teachers provided data about 
their use of their nominated strategies and reactions to the professional learning. 
An interview with the principal also occurred in Stage 5. In addition, student 
learning was analysed by comparing NAPLAN results for the Year 7 and 9 
students in 2009 with their Year 5 and Year 7 results respectively in 2007 aligned 
with the corresponding New South Wales data. 

Results and Discussion 

The results are reported in two sections. The first looks at the preferred teaching 
strategies identified and used by the teachers. These data confirm pedagogical 
practices and identify opportunities for teacher change supported by the 
professional learning model. The second section reports on student learning. 
Given there was only two months of teacher implementation between the 
professional conversations and the NAPLAN test, these data are seen as some 
indicator of the professional learning model's success, but not in any way 
conclusive on its own. 

Teaching Strategies 

Teaching strategies data were collected in Stages 2, 3, 4 and 5. Stage 4 
questionnaire and interview data are reported before Stage 3 lesson observation 
data. This order allows for a better comparison of the observations against the 
teachers' reporting. 

Stage 2 

During the professional learning discussions, the teachers reported giving their 
students practise on NAPLAN type items before the tests. However, there was 
no use of actual school data to inform their planning and practice to support 
student learning, or approaches to build desired understanding in their general 
teaching. When each pre-test item was discussed, teachers were asked to 
estimate the proportion of the school cohort correctly answering each item. They 
tended to overestimate and were frequently surprised by the low proportion of 
correct responses. 
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From looking at the mathematics involved in the identified areas and the 
incorrect answers chosen by students, the teachers and authors chose eight 
strategies as potentially useful for improving students' mathematics and 
numeracy proficiency These strategies contained a mix of general teaching 
strategies and some which are appropriate when conducting class discussions 
based around NAPLAN style items. The teachers indicated during the 
professional learning discussions in Stage 2 that they intended to focus on the 
areas of concern and use strategies to address these from the professional 
learning day not only in their general teaching, but also to use NAPLAN items 
as stimuli for constructive class discussion. 

Stage 4 

After implementation, teachers completed a short questionnaire where they 
ranked the strategies in their preferred order of usefulness. Table 2 shows the 
results from the eight teachers who responded to the questionnaire. Scores were 
calculated by assigning 1 to the first choice, 2 to the second choice and so on, 
hence the lowest score indicates the most preferred strategy and the highest score 
indicates the least preferred (scores could range from 8 to 64). 

Their ranking must be interpreted realising they may not have tried some at 
all and only chose from the specific strategies they did implement. None the less, 
the attractiveness of the ones they did choose to try is a factor in determining 
effective strategies, which promote good pedagogy and are seen as comfortable 
for use by teachers. 

Table 2 

Preferred strategies as selected by the teachers to address students' numeracy learning 


needs 

Strategy Score 

1. Promoting interpretation of context-based mathematics 20 

questions using Newman's error analysis questions 

2. Developing efficient mental computation strategies 29 

3. Using estimation strategies with all questions 36 

4. Eliminating possibilities in multiple choice questions 41 

5. Checking reasonableness of answers 43 

6. Developing visualisation strategies in geometry 47 

(2D to 3D and 3D to 2D representations) 

7. Identifying irrelevant information in mathematics 52 

questions and problems 

8. Developing strategies for answering open-ended questions 58 
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Newman's questions and mental computation emerged as the most popular 
choices with 7 teachers ranking Newman's in the top 3. The final teacher ranked 
it last - so among 7 of the teachers, the preference was strong. Teachers' comments 
revealed some believed they were already using such strategies. For example: 

The majority of the strategies I already used prior to the PD except for the 
Newman's Method. 

Others found the opportunity to consider new approaches was beneficial to both 
their teaching and student learning as shown by the comments below from three 
different teachers. 

Identified their need for mental computation and to read all the question. 

I found the Newman's questions are very useful. I went through that with all 
my classes. 

Newman's strategies - worked - ensuring read all of the question. 

Three teachers' comments suggest their knowledge and understanding of the 
potential of NAPLAN items and data have improved: 

It gives me an idea of which kind of questions students found hard so I would 
focus more on those areas. 

Next year I intend to show students a variety of strategies for approaching the 
numeracy tests. I will also target some specific areas of knowledge that students 
in the past have had difficulties with. 

The pre-test identified common areas of weakness in my class. Common 
misconceptions were easily identified by the alternate choices students made 
when choosing the answer. 

Professional dialogue between teachers and the researchers enabled the 
identification of a range of strategies for implementation in classrooms, an 
approach acknowledged as successful by the following three teachers' 
comments: 

It was good to gather with colleagues and to discuss alternate teaching 
strategies. 

It was especially good to get the chance to do practical maths questions and be 
the "student" ourselves. 

Focusing on mental computation, visualisation, Newman's as part of each unit, 
from beginning of the year - encouraging this as a normal part of doing Maths. 

Even though teachers indicated they already used some of the teaching strategies 
in regular lessons, their awareness of the strategies and ability to identify when 
they were using them increased. Further, they had not used them as a focus for 
supporting NAPLAN preparation nor in taking items and through these 
strategies making them a source of constructive class discussion rather than 
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right/wrong drill and practice. The data here show they were still using some of 
the learning from the professional conversations three months after the 
NAPLAN tests. 

Stage 3 

Table 3 shows the strategies identified in the professional learning conversation, 
which were planned for and actually used by the teachers in the eight observed 
lesson. Some teachers used more than one strategy. 

The data set here is not large but still allows for some inference about the 
classroom practices of the participating teachers. The top two strategies 
(Newman's analysis and mental computation) figured prominently but a specific 
focus on estimation did not. All three teachers who used Newman's analysis 
actually went through the steps with the class. Visualisation, though not an 
original popular choice, was used as the basis for three of the lessons. The 
specific test strategy of eliminating possibilities in multiple choice questions was 
planned but not widely used indicting lessons became more involved with the 
mathematics and appropriate procedures rather than test based strategies. As 
one teacher said to her class: 

Does the answer actually fit the question? Have confidence in your ability. 
Table 3 


Teachers' planned and observed aimed at addressing students' numeracy learning needs 



Strategy 

Planned 

Observed 

1 . 

Promoting interpretation of context-based 
mathematics questions using Newman's error 
analysis questions 

3 

3 

2. 

Developing efficient mental computation strategies 

2 

2 

3. 

Using estimation strategies with all questions 

0 

0 

4. 

Eliminating possibilities in multiple choice questions 

3 

1 

5. 

Checking reasonableness of answers 

1 

2 

6. 

Developing visualisation strategies in geometry 
(2D to 3D and 3D to 2D representations) 

3 

3 

7. 

Identifying irrelevant information in mathematics 
questions and problems 

1 

0 

8. 

Developing strategies for answering open-ended 
questions 

0 

0 


Four of the lessons involved NAPLAN items as a source of class discussion and 
group work. In all these lessons, teaching went beyond right/wrong answers 
and looked at procedures. Three involved group work while one was more 
teacher centred. The visualisation lessons were three of the four, which did not 
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use NAPLAN items. The teachers chose other activities to involve groups of 
students building objects given specific properties (for example, can you build 
the shape which looks like this from the front and has the most or least number 
of cubes). In one visualisation lesson, one group was noted as definitely not 
being engaged. The level of student engagement was commented on positively 
in six of the other seven lessons. 

In summary, the teachers in all eight lessons planned for one of the identified 
strategies and in seven of the lessons implemented one or more. In all lessons, the 
focus was on procedures and strategies, not just right answers. Newman's 
analysis appears to have provided a new lens for dealing with mathematics in a 
context. The focus on mental computation supported student thinking rather 
than memory based approaches. 

Stage 5 

The data from stage 5 gave some mixed messages. Table 4 shows the eight 
teachers responses about effects of involvement in the project in September 2010. 

These results suggest the project had some effect on the teachers over a year 
after participation but the effect does not seem emphatic. Further, when asked 
what they did differently now as a result of participation, five of the eight said 
'nothing really'. The three who nominated some change identified 'targeted review 
of questions students found difficult in previous years'; 'problem solving including 
Newman'; 'start with 5 questions (Naplan style)'. Except for Newman, these changes 
do not reflect the intended focus on strategies and the mathematics involved. 


Table 4 

Teachers' responses about effects of involvement in September 2010 


Statement 

Agree 

Disagree 

My involvement in the NAPLAN project has impacted 
on the strategies I use in my general mathematics 
teaching 

6 

2 

Because of the project, I felt more confident in preparing 
my classes for NAPLAN this year 

6 

2 

Since the NAPLAN project, the classroom environment 
in mathematics lessons promotes students' willingness 
to engage more with word problems 

5 

3 

Because of the project, I am more aware of the types of 
errors and misconceptions students have in learning 
mathematics 

5 

3 


However, when asked about the strategies, which had been identified in Stage 4, 
the eight teachers indicated sustained substantial use as shown in Table 5. 
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Table 5 

Frequency of use of identified strategies aimed at addressing students' numeracy learning 
needs 



Strategy 

Regularly 

(weekly) 

Sometimes 

(monthly) 

Never 

1 . 

Promoting interpretation of context- 
based mathematics questions using 
Newman's error analysis questions 

3 

5 

0 

2. 

Developing efficient mental 
computation strategies 

7 

1 

0 

3. 

Using estimation strategies with all 
questions 

7 

1 

0 

4. 

Eliminating possibilities in multiple 
choice questions 

2 

6 

0 

5. 

Checking reasonableness of answers 

6 

1 

1 

6. 

Developing visualisation strategies in 
geometry (2D to 3D and 3D to 2D 
representations) 

2 

6 

0 

7. 

Identifying irrelevant information in 
mathematics questions and problems 

1 

5 

0 

8. 

Developing strategies for answering 
open-ended questions 

5 

3 

0 


The frequency of use is consistent with both the nominated preference of strategy 
in Stage 4 and observed strategies in Stage 3. Estimation here matches Stage 4 
nominations and suggests the Stage 3 non-observation of estimation was just a 
chance occurrence. Newman figured less regularly which can be explained by 
the strategy being very specific to contextualised questions which may not be a 
focus in class much of the time. Checking reasonableness, providing irrelevant 
information and open-ended questions are more prominent than in the earlier 
stages. The high level reported for mental computation and estimation shows a 
positive shift to engaging with working mathematically rather than with rote 
routine procedures. 

The identification of specific strategies suggests more had been taken from 
involvement in the professional learning conversations than was indicated by the 
questionnaire. Interviews with the eight teachers confirm the stronger influence 
of the professional learning model. One teacher commented that 'PD offered was 
an intense time' - this included visits for lesson observation as well. Reflecting 
on changes to their own practices brought comments like: 

Personally use Newman's method as it works well 

Too much and too little information encourages students to think 
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Hands on resources used including centicubes 

Newman's all the time in all subject areas (including RE) 

Logic questions such as are all squares parallelograms? Are all parallelograms 
squares? 

Other comments also indicated that the teachers were making more use of data 
and trends in the data. For example, all the teachers identified for themselves 
that tables and graphs as an area needing attention. 

The interview with the principal indicated she was very pleased with the 
whole professional model saying there was evidence of a positive change in 
classroom practice. She cited an intensive intervention to work on school 
programs as one example, but more importantly their own awareness of 
strategies to use and a heightened consciousness and control of their use of these 
strategies. In addition, she acknowledged teachers increased understanding of 
ways to use data to inform decision-making. She stated: 

The project has energized them and got the discussion going. There is more 
energy and discussion around teaching and learning. The teachers now seem to 
have the language to talk about these things. Staff members were ignoring 
NAPLAN but now they are starting to engage with it. We need to have 
strategies and practices based on, and informed by, data. 

While mixed, the Stage 5 data suggest teachers are more aware of ways to 
implement different pedagogical practices in their classroom even though they 
may be of the opinion that they were doing so all along. In particular, the 
language they use to describe their practice would seem to indicate that they 
have in fact moved to a higher level of awareness about their own practice and 
the potential of evidence to inform their planning and programming. 

Student learning 

Student data from 2007 and 2009 at the sample school for each student were 
compared to the total New South Wales data. The New South Wales data set was 
readily available with the schools data and was seen as an appropriate standard 
to use for comparison. The group of students used in the comparisons was 
exactly the same group in both 2007 and 2009. The mean gain for each group was 
calculated by averaging the entire individual gains. The data need to be 
interpreted realising that an expected mean improvement from Years 5 to 7 is 50 
points and Years 7 to 9 is 40 points and that the professional learning model 
intervention only occurred for two months prior to NAPLAN in 2009. The 
impacts of other unknown factors cannot be ignored. 

To address the impact of other factors, Cohen's coefficient for effect size has 
been calculated for each group. Effect size for statistically significant findings 
attempts to "quantify the importance or substantive influence of the mean 
differences observed" (Kline, 2004, p. 132). Tables 6 and 7 show the results of 
comparing the mean gains using a one tailed t-test along with the associated 
Cohen's coefficient. 
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Table 6 


Year 7 mean gains for the school compared to the NSW means from 2007 to 2009 



Yr 5-7 


p value 


2007 - 2009 

N=123 

t value 

1 tail 

Cohen's d 

Sample School Mean Gain 
NSW Mean Gain 

66.08 

55.2 

2.508 

0.008 

0.23 


The results in Table 6 show that the gains by the sample school compared to the 
state are significant at the 1% level for Year 7. The results in Table 7 show that the 
gains by the sample school compared to the state are significant at the 2% level 
for Year 9. 


Table 7 

Year 9 mean gains for the school compared to the NSW means from 2007 to 2009 



Yr 5-7 


p value 


2007 - 2009 

N=126 

t value 

1 tail 

Cohen's d 

Sample School Mean Gain 

45.5 

2.138 

0.017 

0.19 

NSW Mean Gain 

38.1 





Cohen's coefficient for effect size (0.23 for Year 7 and 0.19 for year 9) supports 
that the intervention alone is not likely to be responsible for the statistically 
significant differences in mean achievements. The sample values of 0.19 and 0.23 
for Cohen's d Effect Size suggest that the mean differences are of a small rather 
than a large substantive difference accounting for a small proportion of the 
variation. A Cohen's d of "0.2 or greater corresponds to a small-sized mean 
difference" (Kline, 2004, p. 132). A coefficient of 0.5 represents a medium effect 
size. This proportion of the variation between the means, however, is statistically 
very unlikely to have occurred by chance alone (Kline, 2004). 

The analyses of the mean differences reported in the NAPLAN comparative 
data do support a positive impact of the professional learning model on student 
learning but the professional learning can only be viewed as one factor impacting 
on the gains. Analysis of the model needs to be much wider than just statistical 
measures of student improvement. 

Another source related to student learning is the teacher comments about 
the students' approach to solving problems and their overall attitude to engaging 
with mathematics. The eight teachers who responded to questions about what 
strategies worked and how these impacted on student attitudes indicated the 
chosen teaching approaches encouraged students to be more confident. Three 
different teacher comments from the eight are: 
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Using a variety of strategies empowers students with ways to better respond to 
set questions in class tests and exams. 

Students gain confidence when they feel that they have been well prepared for 
tests and they perceive that it is important they try their best. They need to get 
used to the language used and the style of questions as well as improving their 
numeracy knowledge. 

Students seem a bit more confident and are more inclined to have a go now. 

Both the quantitative and qualitative data support an improvement in learning 
though this improvement cannot be directly linked to involvement in the project 
alone. Importantly, students' experiences were positive and they gained 
confidence in tackling NAPLAN-style questions. 

Conclusions 

There is evidence that engagement in the professional learning model by 
teachers coincided with some positive student learning outcomes. The school 
thus saw the project as successful. The mix of using clearly identified strategies 
in general class teaching with NAPLAN items as a stimulus for discussion, 
appears to be an effective pedagogical combination. The results here are 
consistent with Martin's (2003) observation that showing students test items and 
discussing strategies for thinking about questions and responses promotes 
student confidence and resilience, and enables a greater sense of student control 
over their learning. In addition, the professional learning model aimed to 
improve the assessment literacy (Gulek, 2003) of teachers, and develop their 
attitudes and beliefs about the potential of using NAPLAN data for planning and 
teaching. Using data to inform teaching certainly became apparent as part of 
teaching practice where no indication of doing so previously was evident. 
However, there is no conclusive evidence about the way the data were used and 
the degree to which such use impacted on teaching practice and student 
learning. 

Overall, teachers' comments (especially ones immediately after the 
implementation) supported the efficacy of the professional learning model. 
Interestingly, teachers' general comments in interviews one year on indicated 
they felt involvement had no real impact on their teaching practice and 
confidence but principal comments and their identification of their teaching 
strategies suggest there were long term changes to their practice. We conclude 
that a professional learning model like this does have a positive impact on 
mathematics learning and teaching but that unless conversations are revisited 
regularly, can mean awareness of the impact is lost in the day-to-day hustle and 
bustle of school life and teacher involvement in a range of initiatives. 

The problem which can arise with high stakes testing where comparative 
results are in the public domain is that the tests become the curriculum and 
teaching strategies become restricted to improving test performance regardless 
of whether any actual learning takes place. Such a position being taken by 
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teachers is understandable, but, as shown in this paper, not the only approach 
available to improving test performance. National testing programs provide 
challenges and opportunities for mathematics teachers. One challenge is to focus 
on the diverse learning needs of students while preparing them for national 
testing early in the school year. 

In the project reported here, a professional learning model was implemented 
with a fair degree of success with two teams of teachers (those teaching Year 7 
and those teaching Year 9), which aimed to turn NAPLAN into a teaching 
resource and a means of taking control of the testing agenda. 

Ideally, the opportunity to form collaborative professional learning teams is 
desirable where some expertise about research into teaching mathematics can be 
accessed. However, the professional learning model described above need not be 
dependent on such external input and can be engaged with individually or in 
small groups within a school. As noted by Perso (2009, p. 11), "teacher reflection 
on student results becomes a powerful tool to guide the teaching of mathematics 
for numeracy by students". In fact, the results indicate more teacher input rather 
than less is desirable. 

The model presented here is not advocating 'teaching to the test', rather it 
supports the notion that there is much to learn from using data available from a 
school's NAPLAN results and items to develop discipline knowledge as well as 
pedagogical content knowledge about important mathematics concepts. Nor 
does the approach presented here advocate national testing as the most desirable 
approach to assessing students' knowledge, skills and understanding. Teachers 
best carry out assessment as they talk to and observe their students (AAMT, 
2008). However, given the reality we face and the fact that many teachers do feel 
pressure to actively prepare their students for the tests, the model presented here 
offers some ideas for a constructive way to do so. Future development of the 
model is therefore indicated and, in particular, the results suggest looking for 
ways to increase long-term positive beliefs, awareness of the impact of the model 
and ownership by teachers. 


References 

AAMT (2008). Position paper on the practice of assessing mathematics learning. Adelaide, SA: 
AAMT. 

Abrams, L. M., Pedula, J. J., & Madaus, G. F. (2003). Views from the classroom: Teachers' 
opinions of statewide testing programs. Theory into Practice, 42(1), 18-29). 

Anderson, J. (2009). Using NAPLAN items to develop students' thinking skills and build 
confidence. The Australian Mathematics Teacher, 65(4), 17-23. 

Curriculum Corporation (2000). Numeracy Benchmarks Years 3, 5 & 7. Carlton, Vic.: Currie. 
Corporation. 

Curriculum Corporation (2006). Statements of learning for mathematics. Melb., Vic: Currie. 
Corporation. 

Department of Education, Training and Youth Affairs (DETYA) (2000), Numeracy, A 
priority for all: Challenges for Australian Schools. Canberra: Commonwealth of 
Australia. 


Pressure to Perform: Reviewing the Use of Data through Professional Learning Conversations 


77 


Dimarco, S. (2009). Crossing the divide between teacher professionalism and national 
testing in middle school mathematics. The Australian Mathematics Teacher, 65(4), 6-10. 

Earl, L. M., & Timperley, H. (2009). Understanding how evidence and learning 
conversations work. In L. M. Earl & H. Timperley (Eds.), Professional Learning 
Conversations: Challenges in Using Evidence for Improvement (pp. 1-12). 
Netherlands: Springer. 

Gould, P., Outhred, L., & Mitchelmore, M. (2006). One-Third is Three-Quarters of One- 
Half. In P. Grootenboer, R. Zevenbergen, & M. Chinnappan, Identities, cultures and 
learning spaces, Volume 1, (pp. 262-270). Proceedings of the 29th Annual Conference 
of the Mathematics Education Research Group of Australasia, Canberra, ACT. 

Gulek, C. (2003). Preparing for high-stakes testing. Theory into Practice, 42(1), 42-50. 

Guskey, T. R. (2002). Professional development and teacher change. Teachers and teaching: 
Theory and practice, 8(3/4), 381-391. 

Kline, R. B. (2004). Beyond significance testing: Reforming data analysis methods in behavioral 
research. American Psychological Association, Washington DC. 

Martin, A. J. (2003). How to motivate your child for school and beyond. Sydney: Bantam. 

Marzano, R. }., Kendall, }. S., & Gaddy, B. B. (1999). Essential knowledge: The debate over how 
American students should know. Aurora, CO: McREL Institute. 

McIntosh, A., Reys, R. E., & Reys, B. J. (1997). Mental computation in the middle grades: 
The importance of thinking strategies. Mathematics Teaching in the Middle School, 2(5), 
322-327. 

Miyasaka, J. R. (2000). A framework for evaluating the validity of test preparation practices. 
Paper presented a the annual meeting of the American Educational Research 
Association, New Orleans. 

Newman, A. (1983). The Newman language of mathematics kit. Sydney, NSW: HBJ. 

Nichols, S. L., & Berliner, D. C. (2005). The inevitable corruption of indicators and educators 
through high-stakes testing. East Lansing, MI: Great Lakes Centre for Education 
Research and Practice. 

Nisbet, S. (2004). The impact of state-wide numeracy testing on the teaching of 
mathematics in primary schools. In M. Homes & A Fuglestad (Eds.), Proceedings of the 
28th Conference of the International Group for the Psychology of Mathematics Education 
(PME) (Vol. 3, pp. 433-440). Norway: PME. 

Perso, T. (2009). Cracking the NAPLAN code: Numeracy in action. The Australian 
Mathematics Teacher, 65(4), 11-16. 

Ryan, J., & Williams, J. (2007). Children's mathematics 4-15: Learning from errors and 
misconceptions. Maidenhead, UK: Open University Press. 

Thomson, S., & Buckley, S. (2009). Informing mathematics pedagogy: TIMSS 2007 Australia 
and the world. Camberwell, Vic.: Australian Council for Educational Research. 

Wilson, M., & Cooney, T. (2002). Mathematics teacher change and development. In G. C. 
Leder, E. Pehkonen, & G. Torner (Eds.), Beliefs: A hidden variable in mathematics 
education? (pp. 127-147). Dordrecht: Kluwer Academic. 

White, A. (2005). Active mathematics in classrooms: Finding out why children make 
mistakes - And doing something to help them. Square One, 15(4), 15-19. 


Authors 

Paul White, School of Education, Australian Catholic University NSW 2135, Australia. 
Email: <paul.white@acu.edu.au> 

Judy Anderson, Faculty of Education and Social Work, The University of Sydney, 
NSW,2006, Australian. Email: <judy.anderson@sydney.edu.au> 



